Uplink congestion mitigation

ABSTRACT

System and techniques for uplink congestion mitigation are described herein. A packet acknowledgement (ACK) rate may be measured in a transmission queue to detect when the ACK rate exceeds a threshold. In response to the ACK rate exceeding the threshold, ACKs may be removed from the transmission queue in accordance with a time-based uplink reduction function. ACKs that remain in the transmission queue after ACKs are removed in accordance with the time-based uplink reduction function may then be transmitted.

TECHNICAL FIELD

Embodiments described herein generally relate to computer networking andmore specifically to uplink congestion mitigation.

BACKGROUND

The transmission control protocol (TCP) is a network communicationstandard at the transport layer of a networking stack. TCP generallyprovides facilities to reliably (e.g., in order, error checked, withre-transmission, etc.) transport data via an internet protocol (IP)between host devices. TCP generally includes establishing connectionsbetween hosts and transferring data over established connections. Thus,TCP may involve greater setup than stateless protocols, such as the userdatagram protocol (UDP). In addition to providing facilities for errorchecking, in-order delivery, and retransmission of data, TCP alsoincludes a number of congestion recognition and mitigation capabilities.

TCP operates at a network layer somewhat higher than the physical (PHY)or media access (MAC) layers. In next generation wireless networks,these PHY and MAC layers are evolving to include more radio bands in awider range of frequencies than have generally been used in the past.Millimeter wave PHY layers are becoming more common with the nextgeneration wireless networks. Millimeter wave radio band communicationsfor next generation wireless networks may provide significant bandwidthand latency benefits. Often, bands are structured such that downlink(e.g., from the network to a device) channels have more bandwidth thanuplink channels, to address likely use cases in which the uplink is usedprimarily to transmit compact query data or acknowledgments (ACKs) fromthe device to the network.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1 is a block diagram of an example of an environment including asystem for uplink congestion mitigation, according to an embodiment.

FIG. 2 is a block diagram of component communications for uplinkcongestion mitigation, according to an embodiment.

FIG. 3 illustrates a flow diagram of an example of a method for uplinkcongestion mitigation, according to an embodiment.

FIG. 4 illustrates a flow diagram of an example of a method for uplinkcongestion mitigation, according to an embodiment.

FIG. 5 illustrates an example domain topology for respective Internet ofThings networks coupled through links to respective gateways, accordingto an embodiment.

FIG. 6 illustrates a cloud computing network in communication with amesh network of Internet of Things devices operating as a fog device atthe edge of the cloud computing network, according to an embodiment.

FIG. 7 illustrates a drawing of a cloud computing network, or cloud, incommunication with a number of Internet of Things devices, according toan embodiment.

FIG. 8 is a block diagram of an example of components that may bepresent in an Internet of Things device, according to an embodiment.

FIG. 9 is a block diagram illustrating an example of a machine uponwhich one or more embodiments may be implemented.

FIG. 10 illustrates an impact of the ‘M’ parameter on TCP e2eperformance, according to an embodiment.

FIG. 11 illustrates an impact of ACK filtering TCP slow-start in ascenario involving multiple TCP flows sharing the same fifth-generationmillimeter wave link, according to an embodiment.

DETAILED DESCRIPTION

Next generation wireless networks (e.g., cellular systems) may providevery high peak data rates (e.g., ten gigabits per second (Gbps)) byusing higher frequency portions of the radio spectrum (e.g., millimeterwave) than were used previously. However, these higher frequency bandsare often highly sensitive to environmental conditions, such as signalinterference from buildings, vehicles, or even atmospheric moisture,leading to disproportionately high path losses when compared to lowerfrequency bands as transceivers move away from each other.

To achieve high data rates expected by users, next generation wirelessnetworks are likely to disproportionality allocate radio resource to thedownlink (DL) (e.g., from network infrastructure to user equipment (UE))at the expense of uplink (UL) allocations. Thus, in accordance with thepeak data rate expectations, fifth generation (5G) cellular networks mayallocate radio resources to achieve twenty Gbps in the downlink and tenGbps in the uplink.

Because TCP uses ACKs to prevent a sender from retransmitting the data,uplink resources are required to send the ACKs in proportion to datareceived in the downlink. A typical IP packet size of a TCP ACK isfifty-two bytes, of which thirty-two bytes are devoted to the TCP headerand twenty bytes are devoted to the IP version four (IPv4) header. If IPversion 6 (IPv6) is used, then the ACK is increased to seventy-two bytesto accommodate the forty byte IPv6 header. By comparing a typical IPpacket size of a TCP segment, 1500 bytes, the uplink data rate requiredfor the target downlink data rate may be derived. However, the TCPreceiver side may not issue a TCP ACK for every TCP segment due todelayed ACKs. In a stream of full-sized segments, there may be an ACKfor every second segment. Typical TCP implementations follow thisbehavior. Subsequently, the required uplink data rate may be derived asfollows:

In IPv4 the required uplink data rate is equal to the downlink

${{peak}\mspace{14mu} {rate}*\frac{52}{2*1500}},$

andIn IPv6 the required uplink data rate is equal to the downlink

${peak}\mspace{14mu} {rate}*{\frac{72}{2*1500}.}$

The table below illustrates a few concrete examples:

Required UL data rate DL peak rate IPv4 IPv6 5 Gbps 86.7 Mbps 120 Mbps20 Gbps 346.7 Mbps  480 Mbps

Due to the possible radio link interruptions for high frequency bands,as well as the disparity between downlink and uplink radio resourceallocations, a situation may arise in which the uplink resources neededto properly convey TCP ACKs are unavailable. In these cases, TCP receiveuplink buffers may fill, increasing latency, or underutilizing radiolink resources.

A technique to mitigate uplink congestion due to heavy TCP ACK trafficis to drop some TCP ACKs. This technique, however, may lead to burst TCPdata traffic arrival and slower growth of the TCP congestion window (asender controlled parameter to reduce downlink data rates) during theTCP slow start phase. Further, discarding TCP ACKs may adversely affectsome TCP flows (e.g., streams) or cause an increase downlink data ratedue to data retransmissions.

To address these TCP ACK filtering issues, a TCP ACK presence rate iscalculated based on uplink grants and transmission queue status todetermine whether uplink congestion is occurring. If congestion isdetected, ACKs are discarded in proportion to the congestion andcontribution by various TCP flows. Thus, if a fifty percent reduction inuplink resources (e.g., fifty percent congestion) is determined toneeded to mitigate the congestion, one out of every two TCP acks isdiscarded (e.g., between six ACKs, the first, third, and fifth arediscarded) to mitigate a collection of retransmissions or transmissionsoccurring at the same time. Moreover, if a first flow has two ACKs and asecond flow has four ACKs, one ACK from the first flow and two from thesecond flow will be removed to maintain proportionality between theflows. Additionally, because TCP ACK filtering results in a net negativeduring the TCP slow start phase of a TCP connection, flows in this slowstart phase will not have any ACKs removed. To further smooth theretransmission traffic flow, discarding TCP ACKs may be smoothed overmultiple uplink resource grants, using consecutive uplink grantintervals to minimize the total number of dropped ACKs within any giventime period while still reducing, or eliminating, uplink congestion.Additional details and examples are provided below.

FIG. 1 is a block diagram of an example of an environment including asystem for uplink congestion mitigation, according to an embodiment. Theenvironment includes a TCP endpoint 125, such as a server, contentprovider, etc. providing downlink data to the device 105 (e.g., a UE,stat (STA), tablet, computer, etc.) via the network infrastructure 120(e.g., an access point (AP), enhanced NodeB (eNb), etc.). The device 105is illustrated within a vehicle 115, the mobility of which may lead togreater uplink congestion issues. In an example, the device 105 may bepart of the vehicle 115.

The device 105 includes a network controller 110 (e.g., NIC, processingcircuitry, etc.) coupled to an antenna to communicate with the TCPendpoint 125 via the network infrastructure 120. To implement uplinkcongestion mitigation, the network controller 110 is arranged to measurea packet ACK rate in a transmission queue to detect when the packet ACKrate exceeds a threshold. This measurement enables the networkcontroller 100 to determine whether uplink congestion is an issue. In anexample, the ACK rate is calculated by

${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$

where R is the ACK rate, T is a measurement window, d is a measurementinterval since a last sample, and r is the instant ACK presence rate. Inan example, the value r is set to 1 if the current UL grant is not bigenough for sending all TCP ACKs in the transmission queue, and 0otherwise.

The network controller 110 is arranged to remove ACKs from thetransmission queue in accordance with a time-based uplink reductionfunction in response to the ACK rate exceeding the threshold. Thetime-based uplink reduction function specifies the manner in which toremove the TCP ACKs in order to achieve uplink congestion mitigationwhile also minimizing the number of ACKs removed within a given timeperiod. In an example, the time-based uplink reduction function removesa percentage of ACKs to meet a percentage that corresponds to thethreshold. Thus, if the threshold is a seventy-five percent congestedthreshold, then the time-based uplink reduction function may designatethe removal of seventy-five percent of the ACKs in the transmissionqueue. In an example, the percentage is expressed as a reduced ratio ofACKS to keep over ACKs to remove. Thus, if the percent is seventy-five,then the reduced ratio is three in four. In an example, ACKs are removedin accordance with the ratio. Using the reduced ratio (e.g., three infour rather than six in eight) maintains a minimal ACK removal for agiven time period. Thus, at three in four, three ACKs are removed andone is transmitted, then three ACKs are removed and one is transmitted,and so on. In an example, the time-based uplink reduction functionincludes a smoothing factor to smooth ACK bursts over multiple uplinkgrant intervals. Smoothing across longer time horizons (here measured inuplink grant intervals) may reduce burst retransmissions, furtheralleviating uplink allocation issues.

In an example, multiple traffic flows (e.g., from applications,operation systems, etc. on the device 105) deposit ACKs into thetransmission queue. In an example, the network controller 110 isarranged to remove ACKs from the multiple traffic flows in proportion toACKs contributed by each of the multiple traffic flows. This prevents asingle application from experiencing disproportionate latency, or delay,due to ACK filtering.

In an example, one of the multiple TCP flows is a TCP slow start flow(e.g., during a TCP slow start phase in establishing the TCP connectionto the TCP endpoint 125). In an example, ACKs from the slow start floware not removed from the transmission queue.

The network controller 110 is arranged to transmit ACKs that remain inthe transmission queue after ACKs are removed in accordance with thetime-based uplink reduction function. In an example, the ACKs aretransmission control protocol (TCP) ACKs. In an example, the ACKS areremoved from the transmission queue is a packet data convergenceprotocol (PDCP) layer of a network stack. In an example, the physicallayer (e.g., radio link) of the network stack is a millimeter wave radiolink.

FIG. 2 is a block diagram of component communications for uplinkcongestion mitigation, according to an embodiment. The illustratednetwork components include a TCP/IP layer 205 (e.g., software orhardware interfacing with operating systems, applications, etc.), apacket data convergence (PDCP) 210 component (e.g., IP block, integratedcircuit, processing circuitry, etc.), a radio link control (RLC) 220component, a PHY/MAC 225 component, and a TCP ACK filter 215.

The TCP ACK filter 215 produces an ACK filtering parameter K1 that isused by the PDCP 210 to filter TCP ACKS before they are communicated tothe RLC 220. To produce K1, the TCP ACK filter 215 accepts the followinginputs

R: the estimated TCP ACK presence rate;

T: the measurement window (e.g. one hundred milliseconds);

d: the measurement interval of the last sample; and

r: the instant TCP ACK presence rate:

-   -   1: the current grant is not enough for all TCP ACK packets in        the transmission queue; and    -   0: the current grant is enough for all TCP ACK packet in the        transmission queue.        and computes the TCP ACK presence rate R, when receiving an        uplink grant, as follows:

$R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\}}{T}$

In an example, TCP ACK filtering will be triggered only if R>Rth, whereRth is a pre-defined threshold (e.g., 0.9). The following measurements,or state variables, may also be used:

-   -   P1[i]: the number of queued ACKs for flow #i, where #i is the UE        port number of the TCP flow;    -   N: number of the flows that have TCP ACKs in the queue (e.g.,        max=2¹⁶ flows);    -   P2: the total number of queued ACKs of all flows;    -   K1[i]: the number of to-be-dropped ACKs for flow i;    -   K2: the total number of to-be-dropped ACKs; and    -   S: the total number of ACKs that may be scheduled in the current        uplink grant;        as described below with respect to FIG. 3.

FIG. 3 illustrates a flow diagram of an example of a method 300 foruplink congestion mitigation, according to an embodiment. The operationsof the method 300 are implemented in electronic hardware, such as thatdescribed above or below (e.g., processing circuitry). The method 300begins by waiting for an event (operation 305), such as the arrival of anew TCP ACK or an uplink (UL) grant (e.g., radio resource allocationfrom network infrastructure).

In response to a new TCP ACK of flow i arrives, the following statevariables are set (operation 330) as follows:

N is incremented by one (e.g., N++)

P1[i] and P2 are each incremented by one (e.g., P1[i]++ and P2++)

and the method 300 waits for another event (operation 305).

In response to an uplink grant, the TCP ACK presence rate R is updated(operation 310) as follows:

$R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\}}{T}$

and R is compared to the configuration variable Rth (decision 315) todetermine whether ACK filtering is triggered. If no, the remaining TCPACKS in the transmission queue will be sent (operation 325). Otherwise,the following state variables are set (operation 320) as follows:

${K\; 2} = {\max \left( {0,{\frac{P\; 2}{M} - S}} \right)}$${{{IF}\mspace{14mu} P\; {1\lbrack i\rbrack}} \leq {S*\frac{M}{N}}},{{{then}\mspace{14mu} K\; {1\lbrack i\rbrack}} = 0}$${{ELSE}\mspace{14mu} K\; {1\lbrack i\rbrack}} = {\min \left( {{{P\lbrack i\rbrack} - 1},{K\; 2*{floor}\mspace{14mu} \left( \frac{P\; {1\lbrack i\rbrack}}{P\; 2} \right)}} \right)}$

and TCP ACKS are filtered according to K1[i]. The IF statementdetermines whether the flow i is in a TCP slow start portion of theconnection and specifies that no ACKs should be removed for that flowwhile in the slow start portion of the connection. M is a configurationvariable used to smooth ACK filtering over multiple uplink grants. Forexample, no ACKs will be discarded (e.g., K2=0), while all the ACKs inthe transmission queue are schedulable in M uplink grant intervals,assuming a grant size of S for each grant.

To ensure fairness, the number of dropped TCP ACKs, K1[i], isproportional to the number of TCP ACKs for each flow,

$\frac{P\; {1\lbrack i\rbrack}}{P\; 2}.$

If, however, a flow has too few ACKs in the transmission queue (e.g.,for flow one,

$\left. {{P\; {1\lbrack 1\rbrack}} < {S*\frac{M}{N}}} \right),$

then no ACKs of the flow are dropped to avoid performance impacts of ACKfiltering during the TCP slow start phase.

The method 300 continues to send remaining TCP ACKs in the uplink grant(operation 325) and updates the state variables. State variable N isdecremented by one (e.g., N−−) when P1[i]=1 and otherwise the statevariables P1[i] and P2 are decremented by one (e.g., P1[i]−− and P2−−)(operation 330) to account for the TCP ACKs removed from thetransmission queue. The method 300 then proceeds to wait for anotherevent (operation 305).

FIG. 4 illustrates a flow diagram of an example of a method 400 foruplink congestion mitigation, according to an embodiment. The operationsof the method 400 are performed by electronic hardware, such as thatdescribed above or below (e.g., processing circuitry).

At operation 405, a packet ACK rate is measured in a transmission queueto detect when the ACK rate exceeds a threshold. In an example, the ACKrate is calculated by

$R = {\frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}.}$

Here, R is the ACK rate, T is a measurement window, d is a measurementinterval since a last sample, and r is a count of ACKs in thetransmission queue.

At operation 410, in response to the ACK rate exceeding the threshold,ACKs are removed from the transmission queue in accordance with atime-based uplink reduction function. In an example, the time-baseduplink reduction function removes a percentage of ACKs to meet apercentage that corresponds to the threshold. In an example, thepercentage is expressed as a reduced ratio of ACKs to keep over ACKs toremove. In an example, ACKs are removed in accordance with the ratio. Inan example, the time-based uplink reduction function includes asmoothing factor to smooth ACK bursts over multiple uplink grantintervals.

In an example, multiple traffic flows deposit ACKs into the transmissionqueue. In an example, removing the ACKs from the transmission queueincludes removing ACKs from the multiple traffic flows in proportion toACKs contributed by each of the multiple traffic flows.

In an example, the ACKs are TCP ACKs. In an example, removing the ACKSfrom the transmission queue is performed at a PDCP layer of a networkstack. In an example, a physical layer of the network stack is amillimeter wave radio link.

In an example, multiple TCP streams deposit ACKs into the transmissionqueue, with one stream being a TCP slow start flow. In an example, ACKsfrom the slow start flow are not removed from the transmission queue.

At operation 415, ACKs that remain in the transmission queue, after ACKsare removed in accordance with the time-based uplink reduction function,are transmitted.

FIG. 5 illustrates an example domain topology for respective Internet ofThings (IoT) networks coupled through links to respective gateways. Theinternet of things (IoT) is a concept in which a large number ofcomputing devices are interconnected to each other and to the Internetto provide functionality and data acquisition at very low levels. Thus,as used herein, an IoT device may include a semiautonomous deviceperforming a function, such as sensing or control, among others, incommunication with other IoT devices and a wider network, such as theInternet.

Often, IoT devices are limited in memory, size, or functionality,allowing larger numbers to be deployed for a similar cost to smallernumbers of larger devices. However, an IoT device may be a smart phone,laptop, tablet, or PC, or other larger device. Further, an IoT devicemay be a virtual device, such as an application on a smart phone orother computing device. IoT devices may include IoT gateways, used tocouple IoT devices to other IoT devices and to cloud applications, fordata storage, process control, and the like.

Networks of IoT devices may include commercial and home automationdevices, such as water distribution systems, electric power distributionsystems, pipeline control systems, plant control systems, lightswitches, thermostats, locks, cameras, alarms, motion sensors, and thelike. The IoT devices may be accessible through remote computers,servers, and other systems, for example, to control systems or accessdata.

The future growth of the Internet and like networks may involve verylarge numbers of IoT devices. Accordingly, in the context of thetechniques discussed herein, a number of innovations for such futurenetworking will address the need for all these layers to growunhindered, to discover and make accessible connected resources, and tosupport the ability to hide and compartmentalize connected resources.Any number of network protocols and communications standards may beused, wherein each protocol and standard is designed to address specificobjectives. Further, the protocols are part of the fabric supportinghuman accessible services that operate regardless of location, time orspace. The innovations include service delivery and associatedinfrastructure, such as hardware and software; security enhancements;and the provision of services based on Quality of Service (QoS) termsspecified in service level and service delivery agreements. As will beunderstood, the use of IoT devices and networks, such as thoseintroduced in FIGS. 5 and 6, present a number of new challenges in aheterogeneous network of connectivity comprising a combination of wiredand wireless technologies.

FIG. 5 specifically provides a simplified drawing of a domain topologythat may be used for a number of Internet of Things (IoT) networkscomprising IoT devices 504, with the IoT networks 556, 558, 560, 562,coupled through backbone links 502 to respective gateways 554. Forexample, a number of IoT devices 504 may communicate with a gateway 554,and with each other through the gateway 554. To simplify the drawing,not every IoT device 504, or communications link (e.g., link 516, 522,528, or 532) is labeled. The backbone links 502 may include any numberof wired or wireless technologies, including optical networks, and maybe part of a local area network (LAN), a wide area network (WAN), or theInternet. Additionally, such communication links facilitate opticalsignal paths among both IoT devices 504 and gateways 554, including theuse of MUXing/deMUXing components that facilitate interconnection of thevarious devices.

The network topology may include any number of types of IoT networks,such as a mesh network provided with the network 556 using Bluetooth lowenergy (BLE) links 522. Other types of IoT networks that may be presentinclude a wireless local area network (WLAN) network 558 used tocommunicate with IoT devices 504 through IEEE 802.11 (Wi-Fi®) links 528,a cellular network 560 used to communicate with IoT devices 504 throughan LTE/LTE-A (4G) or 5G cellular network, and a low-power wide area(LPWA) network 562, for example, a LPWA network compatible with theLoRaWan specification promulgated by the LoRa alliance, or a IPv6 overLow Power Wide-Area Networks (LPWAN) network compatible with aspecification promulgated by the Internet Engineering Task Force (IETF).Further, the respective IoT networks may communicate with an outsidenetwork provider (e.g., a tier 2 or tier 3 provider) using any number ofcommunications links, such as an LTE cellular link, an LPWA link, or alink based on the IEEE 802.15.4 standard, such as Zigbee®. Therespective IoT networks may also operate with use of a variety ofnetwork and internet application protocols such as ConstrainedApplication Protocol (CoAP). The respective IoT networks may also beintegrated with coordinator devices that provide a chain of links thatforms cluster tree of linked devices and networks.

Each of these IoT networks may provide opportunities for new technicalfeatures, such as those as described herein. The improved technologiesand networks may enable the exponential growth of devices and networks,including the use of IoT networks into as fog devices or systems. As theuse of such improved technologies grows, the IoT networks may bedeveloped for self-management, functional evolution, and collaboration,without needing direct human intervention. The improved technologies mayeven enable IoT networks to function without centralized controlledsystems. Accordingly, the improved technologies described herein may beused to automate and enhance network management and operation functionsfar beyond current implementations.

In an example, communications between IoT devices 504, such as over thebackbone links 502, may be protected by a decentralized system forauthentication, authorization, and accounting (AAA). In a decentralizedAAA system, distributed payment, credit, audit, authorization, andauthentication systems may be implemented across interconnectedheterogeneous network infrastructure. This allows systems and networksto move towards autonomous operations. In these types of autonomousoperations, machines may even contract for human resources and negotiatepartnerships with other machine networks. This may allow the achievementof mutual objectives and balanced service delivery against outlined,planned service level agreements as well as achieve solutions thatprovide metering, measurements, traceability and trackability. Thecreation of new supply chain structures and methods may enable amultitude of services to be created, mined for value, and collapsedwithout any human involvement.

Such IoT networks may be further enhanced by the integration of sensingtechnologies, such as sound, light, electronic traffic, facial andpattern recognition, smell, vibration, into the autonomous organizationsamong the IoT devices. The integration of sensory systems may allowsystematic and autonomous communication and coordination of servicedelivery against contractual service objectives, orchestration andquality of service (QoS) based swarming and fusion of resources. Some ofthe individual examples of network-based resource processing include thefollowing.

The mesh network 556, for instance, may be enhanced by systems thatperform inline data-to-information transforms. For example, self-formingchains of processing resources comprising a multi-link network maydistribute the transformation of raw data to information in an efficientmanner, and the ability to differentiate between assets and resourcesand the associated management of each. Furthermore, the propercomponents of infrastructure and resource based trust and serviceindices may be inserted to improve the data integrity, quality,assurance and deliver a metric of data confidence.

The WLAN network 558, for instance, may use systems that performstandards conversion to provide multi-standard connectivity, enablingIoT devices 504 using different protocols to communicate. Furthersystems may provide seamless interconnectivity across a multi-standardinfrastructure comprising visible Internet resources and hidden Internetresources.

Communications in the cellular network 560, for instance, may beenhanced by systems that offload data, extend communications to moreremote devices, or both. The LPWA network 562 may include systems thatperform non-Internet protocol (IP) to IP interconnections, addressing,and routing. Further, each of the IoT devices 504 may include theappropriate transceiver for wide area communications with that device.Further, each IoT device 504 may include other transceivers forcommunications using additional protocols and frequencies. This isdiscussed further with respect to the communication environment andhardware of an IoT processing device depicted in FIGS. 7 and 8.

Finally, clusters of IoT devices may be equipped to communicate withother IoT devices as well as with a cloud network. This may allow theIoT devices to form an ad-hoc network between the devices, allowing themto function as a single device, which may be termed a fog device. Thisconfiguration is discussed further with respect to FIG. 6 below.

FIG. 6 illustrates a cloud computing network in communication with amesh network of IoT devices (devices 602) operating as a fog device atthe edge of the cloud computing network. The mesh network of IoT devicesmay be termed a fog 620, operating at the edge of the cloud 600. Tosimplify the diagram, not every IoT device 602 is labeled.

The fog 620 may be considered to be a massively interconnected networkwherein a number of IoT devices 602 are in communications with eachother, for example, by radio links 622. As an example, thisinterconnected network may be facilitated using an interconnectspecification released by the Open Connectivity Foundation™ (OCF). Thisstandard allows devices to discover each other and establishcommunications for interconnects. Other interconnection protocols mayalso be used, including, for example, the optimized link state routing(OLSR) Protocol, the better approach to mobile ad-hoc networking(B.A.T.M.A.N.) routing protocol, or the OMA Lightweight M2M (LWM2M)protocol, among others.

Three types of IoT devices 602 are shown in this example, gateways 604,data aggregators 626, and sensors 628, although any combinations of IoTdevices 602 and functionality may be used. The gateways 604 may be edgedevices that provide communications between the cloud 600 and the fog620, and may also provide the backend process function for data obtainedfrom sensors 628, such as motion data, flow data, temperature data, andthe like. The data aggregators 626 may collect data from any number ofthe sensors 628, and perform the back end processing function for theanalysis. The results, raw data, or both may be passed along to thecloud 600 through the gateways 604. The sensors 628 may be full IoTdevices 602, for example, capable of both collecting data and processingthe data. In some cases, the sensors 628 may be more limited infunctionality, for example, collecting the data and allowing the dataaggregators 626 or gateways 604 to process the data.

Communications from any IoT device 602 may be passed along a convenientpath (e.g., a most convenient path) between any of the IoT devices 602to reach the gateways 604. In these networks, the number ofinterconnections provide substantial redundancy, allowing communicationsto be maintained, even with the loss of a number of IoT devices 602.Further, the use of a mesh network may allow IoT devices 602 that arevery low power or located at a distance from infrastructure to be used,as the range to connect to another IoT device 602 may be much less thanthe range to connect to the gateways 604.

The fog 620 provided from these IoT devices 602 may be presented todevices in the cloud 600, such as a server 606, as a single devicelocated at the edge of the cloud 600, e.g., a fog device. In thisexample, the alerts coming from the fog device may be sent without beingidentified as coming from a specific IoT device 602 within the fog 620.In this fashion, the fog 620 may be considered a distributed platformthat provides computing and storage resources to perform processing ordata-intensive tasks such as data analytics, data aggregation, andmachine-learning, among others.

In some examples, the IoT devices 602 may be configured using animperative programming style. e.g., with each IoT device 602 having aspecific function and communication partners. However, the IoT devices602 forming the fog device may be configured in a declarativeprogramming style, allowing the IoT devices 602 to reconfigure theiroperations and communications, such as to determine needed resources inresponse to conditions, queries, and device failures. As an example, aquery from a user located at a server 606 about the operations of asubset of equipment monitored by the IoT devices 602 may result in thefog 620 device selecting the IoT devices 602, such as particular sensors628, needed to answer the query. The data from these sensors 628 maythen be aggregated and analyzed by any combination of the sensors 628,data aggregators 626, or gateways 604, before being sent on by the fog620 device to the server 606 to answer the query. In this example, IoTdevices 602 in the fog 620 may select the sensors 628 used based on thequery, such as adding data from flow sensors or temperature sensors.Further, if some of the IoT devices 602 are not operational, other IoTdevices 602 in the fog 620 device may provide analogous data, ifavailable.

In other examples, the operations and functionality described above maybe embodied by a IoT device machine in the example form of an electronicprocessing system, within which a set or sequence of instructions may beexecuted to cause the electronic processing system to perform any one ofthe methodologies discussed herein, according to an example embodiment.The machine may be an IoT device or an IoT gateway, including a machineembodied by aspects of a personal computer (PC), a tablet PC, a personaldigital assistant (PDA), a mobile telephone or smartphone, or anymachine capable of executing instructions (sequential or otherwise) thatspecify actions to be taken by that machine. Further, while only asingle machine may be depicted and referenced in the example above, suchmachine shall also be taken to include any collection of machines thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein.Further, these and like examples to a processor-based system shall betaken to include any set of one or more machines that are controlled byor operated by a processor (e.g., a computer) to individually or jointlyexecute instructions to perform any one or more of the methodologiesdiscussed herein.

FIG. 7 illustrates a drawing of a cloud computing network, or cloud 700,in communication with a number of Internet of Things (IoT) devices. Thecloud 700 may represent the Internet, or may be a local area network(LAN), or a wide area network (WAN), such as a proprietary network for acompany. The IoT devices may include any number of different types ofdevices, grouped in various combinations. For example, a traffic controlgroup 706 may include IoT devices along streets in a city. These IoTdevices may include stoplights, traffic flow monitors, cameras, weathersensors, and the like. The traffic control group 706, or othersubgroups, may be in communication with the cloud 700 through wired orwireless links 708, such as LPWA links, optical links, and the like.Further, a wired or wireless sub-network 712 may allow the IoT devicesto communicate with each other, such as through a local area network, awireless local area network, and the like. The IoT devices may useanother device, such as a gateway 710 or 728 to communicate with remotelocations such as the cloud 700; the IoT devices may also use one ormore servers 730 to facilitate communication with the cloud 700 or withthe gateway 710. For example, the one or more servers 730 may operate asan intermediate network node to support a local edge cloud or fogimplementation among a local area network. Further, the gateway 728 thatis depicted may operate in a cloud-to-gateway-to-many edge devicesconfiguration, such as with the various IoT devices 714, 720, 724 beingconstrained or dynamic to an assignment and use of resources in thecloud 700.

Other example groups of IoT devices may include remote weather stations714, local information terminals 716, alarm systems 718, automatedteller machines 720, alarm panels 722, or moving vehicles, such asemergency vehicles 724 or other vehicles 726, among many others. Each ofthese IoT devices may be in communication with other IoT devices, withservers 704, with another IoT fog device or system (not shown, butdepicted in FIG. 6), or a combination therein. The groups of IoT devicesmay be deployed in various residential, commercial, and industrialsettings (including in both private or public environments).

As may be seen from FIG. 7, a large number of IoT devices may becommunicating through the cloud 700. This may allow different IoTdevices to request or provide information to other devices autonomously.For example, a group of IoT devices (e.g., the traffic control group706) may request a current weather forecast from a group of remoteweather stations 714, which may provide the forecast without humanintervention. Further, an emergency vehicle 724 may be alerted by anautomated teller machine 720 that a burglary is in progress. As theemergency vehicle 724 proceeds towards the automated teller machine 720,it may access the traffic control group 706 to request clearance to thelocation, for example, by lights turning red to block cross traffic atan intersection in sufficient time for the emergency vehicle 724 to haveunimpeded access to the intersection.

Clusters of IoT devices, such as the remote weather stations 714 or thetraffic control group 706, may be equipped to communicate with other IoTdevices as well as with the cloud 700. This may allow the IoT devices toform an ad-hoc network between the devices, allowing them to function asa single device, which may be termed a fog device or system (e.g., asdescribed above with reference to FIG. 6).

FIG. 8 is a block diagram of an example of components that may bepresent in an IoT device 850 for implementing the techniques describedherein. The IoT device 850 may include any combinations of thecomponents shown in the example or referenced in the disclosure above.The components may be implemented as ICs, portions thereof, discreteelectronic devices, or other modules, logic, hardware, software,firmware, or a combination thereof adapted in the IoT device 850, or ascomponents otherwise incorporated within a chassis of a larger system.Additionally, the block diagram of FIG. 8 is intended to depict ahigh-level view of components of the IoT device 850. However, some ofthe components shown may be omitted, additional components may bepresent, and different arrangement of the components shown may occur inother implementations.

The IoT device 850 may include a processor 852, which may be amicroprocessor, a multi-core processor, a multithreaded processor, anultra-low voltage processor, an embedded processor, or other knownprocessing element. The processor 852 may be a part of a system on achip (SoC) in which the processor 852 and other components are formedinto a single integrated circuit, or a single package, such as theEdison™ or Galileo™ SoC boards from Intel. As an example, the processor852 may include an Intel® Architecture Core™ based processor, such as aQuark™, an Atom™, an i3, an i5, an i7, or an MCU-class processor, oranother such processor available from Intel® Corporation, Santa Clara,Calif. However, any number other processors may be used, such asavailable from Advanced Micro Devices, Inc. (AMD) of Sunnyvale, Calif.,a MIPS-based design from MIPS Technologies, Inc. of Sunnyvale, Calif.,an ARM-based design licensed from ARM Holdings, Ltd. or customerthereof, or their licensees or adopters. The processors may includeunits such as an A5-A7 processor from Apple® Inc., a Snapdragon™processor from Qualcomm® Technologies, Inc., or an OMAP™ processor fromTexas Instruments, Inc.

The processor 852 may communicate with a system memory 854 over aninterconnect 856 (e.g., a bus). Any number of memory devices may be usedto provide for a given amount of system memory. As examples, the memorymay be random access memory (RAM) in accordance with a Joint ElectronDevices Engineering Council (JEDEC) design such as the DDR or mobile DDRstandards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In variousimplementations the individual memory devices may be of any number ofdifferent package types such as single die package (SDP), dual diepackage (DDP) or quad die package (Q17P). These devices, in someexamples, may be directly soldered onto a motherboard to provide a lowerprofile solution, while in other examples the devices are configured asone or more memory modules that in turn couple to the motherboard by agiven connector. Any number of other memory implementations may be used,such as other types of memory modules, e.g., dual inline memory modules(DIMMs) of different varieties including but not limited to microDIMMsor MiniDIMMs.

To provide for persistent storage of information such as data,applications, operating systems and so forth, a storage 858 may alsocouple to the processor 852 via the interconnect 856. In an example thestorage 858 may be implemented via a solid state disk drive (SSDD).Other devices that may be used for the storage 858 include flash memorycards, such as SD cards, microSD cards, xD picture cards, and the like,and USB flash drives. In low power implementations, the storage 858 maybe on-die memory or registers associated with the processor 852.However, in some examples, the storage 858 may be implemented using amicro hard disk drive (HDD). Further, any number of new technologies maybe used for the storage 858 in addition to, or instead of, thetechnologies described, such resistance change memories, phase changememories, holographic memories, or chemical memories, among others.

The components may communicate over the interconnect 856. Theinterconnect 856 may include any number of technologies, includingindustry standard architecture (ISA), extended ISA (EISA), peripheralcomponent interconnect (PCI), peripheral component interconnect extended(PCIx), PCI express (PCIe), or any number of other technologies. Theinterconnect 856 may be a proprietary bus, for example, used in a SoCbased system. Other bus systems may be included, such as an I2Cinterface, an SPI interface, point to point interfaces, and a power bus,among others.

The interconnect 856 may couple the processor 852 to a mesh transceiver862, for communications with other mesh devices 864. The meshtransceiver 862 may use any number of frequencies and protocols, such as2.4 Gigahertz (GHz) transmissions under the IEEE 802.15.4 standard,using the Bluetooth® low energy (BLE) standard, as defined by theBluetooth® Special Interest Group, or the ZigBee® standard, amongothers. Any number of radios, configured for a particular wirelesscommunication protocol, may be used for the connections to the meshdevices 864. For example, a WLAN unit may be used to implement Wi-Fi™communications in accordance with the Institute of Electrical andElectronics Engineers (IEEE) 802.11 standard. In addition, wireless widearea communications, e.g., according to a cellular or other wirelesswide area protocol, may occur via a WWAN unit.

The mesh transceiver 862 may communicate using multiple standards orradios for communications at different range. For example, the IoTdevice 850 may communicate with close devices, e.g., within about 10meters, using a local transceiver based on BLE, or another low powerradio, to save power. More distant mesh devices 864, e.g., within about50 meters, may be reached over ZigBee or other intermediate powerradios. Both communications techniques may take place over a singleradio at different power levels, or may take place over separatetransceivers, for example, a local transceiver using BLE and a separatemesh transceiver using ZigBee.

A wireless network transceiver 866 may be included to communicate withdevices or services in the cloud 800 via local or wide area networkprotocols. The wireless network transceiver 866 may be a LPWAtransceiver that follows the IEEE 802.15.4, or IEEE 802.15.4g standards,among others. The IoT device 850 may communicate over a wide area usingLoRaWAN™ (Long Range Wide Area Network) developed by Semtech and theLoRa Alliance. The techniques described herein are not limited to thesetechnologies, but may be used with any number of other cloudtransceivers that implement long range, low bandwidth communications,such as Sigfox, and other technologies. Further, other communicationstechniques, such as time-slotted channel hopping, described in the IEEE802.15.4e specification may be used.

Any number of other radio communications and protocols may be used inaddition to the systems mentioned for the mesh transceiver 862 andwireless network transceiver 866, as described herein. For example, theradio transceivers 862 and 866 may include an LTE or other cellulartransceiver that uses spread spectrum (SPA/SAS) communications forimplementing high speed communications. Further, any number of otherprotocols may be used, such as Wi-Fi®networks for medium speedcommunications and provision of network communications.

The radio transceivers 862 and 866 may include radios that arecompatible with any number of 3GPP (Third Generation PartnershipProject) specifications, notably Long Term Evolution (LTE), Long TermEvolution-Advanced (LTE-A), and Long Term Evolution-Advanced Pro (LTE-APro). It may be noted that radios compatible with any number of otherfixed, mobile, or satellite communication technologies and standards maybe selected. These may include, for example, any Cellular Wide Arearadio communication technology, which may include e.g. a 5th Generation(5G) communication systems, a Global System for Mobile Communications(GSM) radio communication technology, a General Packet Radio Service(GPRS) radio communication technology, or an Enhanced Data Rates for GSMEvolution (EDGE) radio communication technology, a UMTS (UniversalMobile Telecommunications System) communication technology, In additionto the standards listed above, any number of satellite uplinktechnologies may be used for the wireless network transceiver 866,including, for example, radios compliant with standards issued by theITU (International Telecommunication Union), or the ETSI (EuropeanTelecommunications Standards Institute), among others. The examplesprovided herein are thus understood as being applicable to various othercommunication technologies, both existing and not yet formulated.

A network interface controller (NIC) 868 may be included to provide awired communication to the cloud 800 or to other devices, such as themesh devices 864. The wired communication may provide an Ethernetconnection, or may be based on other types of networks, such asController Area Network (CAN), Local Interconnect Network (LIN),DeviceNet, ControlNet, Data Highway+, PROFIBUS, or PROFINET, among manyothers. An additional NIC 868 may be included to allow connect to asecond network, for example, a NIC 868 providing communications to thecloud over Ethernet, and a second NIC 868 providing communications toother devices over another type of network.

The interconnect 856 may couple the processor 852 to an externalinterface 870 that is used to connect external devices or subsystems.The external devices may include sensors 872, such as accelerometers,level sensors, flow sensors, optical light sensors, camera sensors,temperature sensors, a global positioning system (GPS) sensors, pressuresensors, barometric pressure sensors, and the like. The externalinterface 870 further may be used to connect the IoT device 850 toactuators 874, such as power switches, valve actuators, an audible soundgenerator, a visual warning device, and the like.

In some optional examples, various input/output (I/O) devices may bepresent within, or connected to, the IoT device 850. For example, adisplay or other output device 884 may be included to show information,such as sensor readings or actuator position. An input device 886, suchas a touch screen or keypad may be included to accept input. An outputdevice 884 may include any number of forms of audio or visual display,including simple visual outputs such as binary status indicators (e.g.,LEDs) and multi-character visual outputs, or more complex outputs suchas display screens (e.g., LCD screens), with the output of characters,graphics, multimedia objects, and the like being generated or producedfrom the operation of the IoT device 850.

A battery 876 may power the IoT device 850, although in examples inwhich the IoT device 850 is mounted in a fixed location, it may have apower supply coupled to an electrical grid. The battery 876 may be alithium ion battery, or a metal-air battery, such as a zinc-air battery,an aluminum-air battery, a lithium-air battery, and the like.

A battery monitor/charger 878 may be included in the IoT device 850 totrack the state of charge (SoCh) of the battery 876. The batterymonitor/charger 878 may be used to monitor other parameters of thebattery 876 to provide failure predictions, such as the state of health(SoH) and the state of function (SoF) of the battery 876. The batterymonitor/charger 878 may include a battery monitoring integrated circuit,such as an LTC4020 or an LTC2990 from Linear Technologies, an ADT7488Afrom ON Semiconductor of Phoenix Ariz., or an IC from the UCD90xxxfamily from Texas Instruments of Dallas, Tex. The batterymonitor/charger 878 may communicate the information on the battery 876to the processor 852 over the interconnect 856. The batterymonitor/charger 878 may also include an analog-to-digital (ADC)convertor that allows the processor 852 to directly monitor the voltageof the battery 876 or the current flow from the battery 876. The batteryparameters may be used to determine actions that the IoT device 850 mayperform, such as transmission frequency, mesh network operation, sensingfrequency, and the like.

A power block 880, or other power supply coupled to a grid, may becoupled with the battery monitor/charger 878 to charge the battery 876.In some examples, the power block 880 may be replaced with a wirelesspower receiver to obtain the power wirelessly, for example, through aloop antenna in the IoT device 850. A wireless battery charging circuit,such as an LTC4020 chip from Linear Technologies of Milpitas, Calif.,among others, may be included in the battery monitor/charger 878. Thespecific charging circuits chosen depend on the size of the battery 876,and thus, the current required. The charging may be performed using theAirfuel standard promulgated by the Airfuel Alliance, the Qi wirelesscharging standard promulgated by the Wireless Power Consortium, or theRezence charging standard, promulgated by the Alliance for WirelessPower, among others.

The storage 858 may include instructions 882 in the form of software,firmware, or hardware commands to implement the techniques describedherein. Although such instructions 882 are shown as code blocks includedin the memory 854 and the storage 858, it may be understood that any ofthe code blocks may be replaced with hardwired circuits, for example,built into an application specific integrated circuit (ASIC).

In an example, the instructions 882 provided via the memory 854, thestorage 858, or the processor 852 may be embodied as a non-transitory,machine readable medium 860 including code to direct the processor 852to perform electronic operations in the IoT device 850. The processor852 may access the non-transitory, machine readable medium 860 over theinterconnect 856. For instance, the non-transitory, machine readablemedium 860 may be embodied by devices described for the storage 858 ormay include specific storage units such as optical disks, flash drives,or any number of other hardware devices. The non-transitory, machinereadable medium 860 may further include, provide, or invoke instructions888 to direct the processor 852 to perform a specific sequence or flowof actions, for example, as described with respect to the flowchart(s)and block diagram(s) of operations and functionality depicted above.

In an example, the instructions 888 on the processor 852 (separately, orin combination with the instructions 888 of the machine readable medium860) may configure execution or operation of a trusted executionenvironment (TEE) 890. In an example, the TEE 890 operates as aprotected area accessible to the processor 852 to enable secure accessto data and secure execution of instructions. Various implementations ofthe TEE 890, and an accompanying secure area in the processor 852 or thememory 854 may be provided, for instance, through use of Intel® SoftwareGuard Extensions (SGX) or ARM® TrustZone® hardware security extensions,Intel® Management Engine (ME), or Intel® Converged SecurityManageability Engine (CSME). Other aspects of security hardening,hardware roots-of-trust, and trusted or protected operations may beimplemented in the device 850 through the TEE 890 and the processor 852.

In further examples, a machine-readable medium also includes anytangible medium that is capable of storing, encoding or carryinginstructions for execution by a machine and that cause the machine toperform any one or more of the methodologies of the present disclosureor that is capable of storing, encoding or carrying data structuresutilized by or associated with such instructions. A “machine-readablemedium” thus may include, but is not limited to, solid-state memories,and optical and magnetic media. Specific examples of machine-readablemedia include non-volatile memory, including but not limited to, by wayof example, semiconductor memory devices (e.g., electricallyprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM)) and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The instructionsembodied by a machine-readable medium may further be transmitted orreceived over a communications network using a transmission medium via anetwork interface device utilizing any one of a number of transferprotocols (e.g., HTTP).

It should be understood that the functional units or capabilitiesdescribed in this specification may have been referred to or labeled ascomponents or modules, in order to more particularly emphasize theirimplementation independence. Such components may be embodied by anynumber of software or hardware forms. For example, a component or modulemay be implemented as a hardware circuit comprising customvery-large-scale integration (VLSI) circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A component or module may also be implemented inprogrammable hardware devices such as field programmable gate arrays,programmable array logic, programmable logic devices, or the like.Components or modules may also be implemented in software for executionby various types of processors. An identified component or module ofexecutable code may, for instance, comprise one or more physical orlogical blocks of computer instructions, which may, for instance, beorganized as an object, procedure, or function. Nevertheless, theexecutables of an identified component or module need not be physicallylocated together, but may comprise disparate instructions stored indifferent locations which, when joined logically together, comprise thecomponent or module and achieve the stated purpose for the component ormodule.

Indeed, a component or module of executable code may be a singleinstruction, or many instructions, and may even be distributed overseveral different code segments, among different programs, and acrossseveral memory devices or processing systems. In particular, someaspects of the described process (such as code rewriting and codeanalysis) may take place on a different processing system (e.g., in acomputer in a data center), than that in which the code is deployed(e.g., in a computer embedded in a sensor or robot). Similarly,operational data may be identified and illustrated herein withincomponents or modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. The components or modules may be passive or active, includingagents operable to perform desired functions.

FIG. 9 illustrates a block diagram of an example machine 900 upon whichany one or more of the techniques (e.g., methodologies) discussed hereinmay perform. Examples, as described herein, may include, or may operateby, logic or a number of components, or mechanisms in the machine 900.Circuitry (e.g., processing circuitry) is a collection of circuitsimplemented in tangible entities of the machine 900 that includehardware (e.g., simple circuits, gates, logic, etc.). Circuitrymembership may be flexible over time. Circuitries include members thatmay, alone or in combination, perform specified operations whenoperating. In an example, hardware of the circuitry may be immutablydesigned to carry out a specific operation (e.g., hardwired). In anexample, the hardware of the circuitry may include variably connectedphysical components (e.g., execution units, transistors, simplecircuits, etc.) including a machine readable medium physically modified(e.g., magnetically, electrically, moveable placement of invariantmassed particles, etc.) to encode instructions of the specificoperation. In connecting the physical components, the underlyingelectrical properties of a hardware constituent are changed, forexample, from an insulator to a conductor or vice versa. Theinstructions enable embedded hardware (e.g., the execution units or aloading mechanism) to create members of the circuitry in hardware viathe variable connections to carry out portions of the specific operationwhen in operation. Accordingly, in an example, the machine readablemedium elements are part of the circuitry or are communicatively coupledto the other components of the circuitry when the device is operating.In an example, any of the physical components may be used in more thanone member of more than one circuitry. For example, under operation,execution units may be used in a first circuit of a first circuitry atone point in time and reused by a second circuit in the first circuitry,or by a third circuit in a second circuitry at a different time.Additional examples of these components with respect to the machine 900follow.

In alternative embodiments, the machine 900 may operate as a standalonedevice or may be connected (e.g., networked) to other machines. In anetworked deployment, the machine 900 may operate in the capacity of aserver machine, a client machine, or both in server-client networkenvironments. In an example, the machine 900 may act as a peer machinein peer-to-peer (P2P) (or other distributed) network environment. Themachine 900 may be a personal computer (PC), a tablet PC, a set-top box(STB), a personal digital assistant (PDA), a mobile telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein, such as cloud computing, software as aservice (SaaS), other computer cluster configurations.

The machine (e.g., computer system) 900 may include a hardware processor902 (e.g., a central processing unit (CPU), a graphics processing unit(GPU), a hardware processor core, or any combination thereof), a mainmemory 904, a static memory (e.g., memory or storage for firmware,microcode, a basic-input-output (BIOS), unified extensible firmwareinterface (UEFI), etc.) 906, and mass storage 908 (e.g., hard drive,tape drive, flash storage, or other block devices) some or all of whichmay communicate with each other via an interlink (e.g., bus) 930. Themachine 900 may further include a display unit 910, an alphanumericinput device 912 (e.g., a keyboard), and a user interface (UI)navigation device 914 (e.g., a mouse). In an example, the display unit910, input device 912 and UI navigation device 914 may be a touch screendisplay. The machine 900 may additionally include a storage device(e.g., drive unit) 908, a signal generation device 918 (e.g., aspeaker), a network interface device 920, and one or more sensors 916,such as a global positioning system (GPS) sensor, compass,accelerometer, or other sensor. The machine 900 may include an outputcontroller 928, such as a serial (e.g., universal serial bus (USB),parallel, or other wired or wireless (e.g., infrared (IR), near fieldcommunication (NFC), etc.) connection to communicate or control one ormore peripheral devices (e.g., a printer, card reader, etc.).

Registers of the processor 902, the main memory 904, the static memory906, or the mass storage 908 may be, or include, a machine readablemedium 922 on which is stored one or more sets of data structures orinstructions 924 (e.g., software) embodying or utilized by any one ormore of the techniques or functions described herein. The instructions924 may also reside, completely or at least partially, within any ofregisters of the processor 902, the main memory 904, the static memory906, or the mass storage 908 during execution thereof by the machine900. In an example, one or any combination of the hardware processor902, the main memory 904, the static memory 906, or the mass storage 908may constitute the machine readable media 922. While the machinereadable medium 922 is illustrated as a single medium, the term “machinereadable medium” may include a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) configured to store the one or more instructions 924.

The term “machine readable medium” may include any medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine 900 and that cause the machine 900 to perform any one ormore of the techniques of the present disclosure, or that is capable ofstoring, encoding or carrying data structures used by or associated withsuch instructions. Non-limiting machine readable medium examples mayinclude solid-state memories, optical media, magnetic media, and signals(e.g., radio frequency signals, other photon based signals, soundsignals, etc.). In an example, a non-transitory machine readable mediumcomprises a machine readable medium with a plurality of particles havinginvariant (e.g., rest) mass, and thus are compositions of matter.Accordingly, non-transitory machine-readable media are machine readablemedia that do not include transitory propagating signals. Specificexamples of non-transitory machine readable media may include:non-volatile memory, such as semiconductor memory devices (e.g.,Electrically Programmable Read-Only Memory (EPROM), ElectricallyErasable Programmable Read-Only Memory (EEPROM)) and flash memorydevices; magnetic disks, such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 924 may be further transmitted or received over acommunications network 926 using a transmission medium via the networkinterface device 920 utilizing any one of a number of transfer protocols(e.g., frame relay, internet protocol (IP), transmission controlprotocol (TCP), user datagram protocol (UDP), hypertext transferprotocol (HTTP), etc.). Example communication networks may include alocal area network (LAN), a wide area network (WAN), a packet datanetwork (e.g., the Internet), mobile telephone networks (e.g., cellularnetworks), Plain Old Telephone (POTS) networks, and wireless datanetworks (e.g., Institute of Electrical and Electronics Engineers (IEEE)802.11 family of standards known as Wi-Fi®, IEEE 802.16 family ofstandards known as WiMax®), IEEE 802.15.4 family of standards,peer-to-peer (P2P) networks, among others. In an example, the networkinterface device 920 may include one or more physical jacks (e.g.,Ethernet, coaxial, or phone jacks) or one or more antennas to connect tothe communications network 926. In an example, the network interfacedevice 920 may include a plurality of antennas to wirelessly communicateusing at least one of single-input multiple-output (SIMO),multiple-input multiple-output (MIMO), or multiple-input single-output(MISO) techniques. The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding orcarrying instructions for execution by the machine 900, and includesdigital or analog communications signals or other intangible medium tofacilitate communication of such software. A transmission medium is amachine readable medium.

FIG. 10 illustrates an impact of the ‘M’ parameter on TCP e2eperformance, according to an embodiment. This performance (e.g.throughput, round trip time) and traffic burstness was observed throughsimulation using the NS3-based 5G simulator (nyuwireless/ns3-mmwave).

CDF (cumulative distribution function) of the burst size, defined as thenumber of TCP data packets triggered by a TCP ACK packet, is used tomeasure burstness. The results show that the adaptive ACK filteringtechnique described above mitigates the UL congestion by reducing delay(round trip time) and increasing throughput. When M=1, the performanceis very close to the case without UL congestion (the optimal point). M=3provides slightly larger delay, but less burstness. Further increasing Mto 9 increases delay but does not reduce burstness anymore in thisexample. Therefore, according to the parameters of this example,configuring M between 1 and 3 provides some benefit.

FIG. 11 illustrates an impact of ACK filtering TCP slow-start in ascenario involving multiple TCP flows sharing the same fifth-generationmillimeter wave link, according to an embodiment. The charts show that,with TCP slow-start detection, ACK filtering for the newly started flowsmay be skipped. As a result, the new flows grow their respective TCPcongestion windows (Cwnds) faster, helping these flows converge faster.

ADDITIONAL NOTES & EXAMPLES

Example 1 is an apparatus for uplink congestion mitigation, theapparatus included in a communication device, the apparatus comprising:a buffer to store a transmission queue; and processing circuitry to:measure a packet acknowledgement (ACK) rate in a transmission queue todetect when the ACK rate exceeds a threshold; remove, in response to theACK rate exceeding the threshold, ACKs from the transmission queue inaccordance with a time-based uplink reduction function; and initiatetransmission ACKs that remain in the transmission queue after ACKs areremoved in accordance with the time-based uplink reduction function.

In Example 2, the subject matter of Example 1 includes, wherein the ACKrate is calculated by

${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$

where R is the ACK rate, T is a measurement window, d is a measurementinterval since a last sample, and r is an instant ACK rate.

In Example 3, the subject matter of Examples 1-2 includes, wherein thetime-based uplink reduction function removes a percentage of ACKs tomeet a percentage that corresponds to the threshold.

In Example 4, the subject matter of Example 3 includes, wherein thepercentage is expressed as a reduced ratio of ACKS to keep over ACKs toremove.

In Example 5, the subject matter of Example 4 includes, wherein ACKs areremoved in accordance with the ratio.

In Example 6, the subject matter of Examples 3-5 includes, wherein thetime-based uplink reduction function includes a smoothing factor tosmooth ACK bursts over multiple uplink grant intervals.

In Example 7, the subject matter of Examples 1-6 includes, whereinmultiple traffic flows deposit ACKs into the transmission queue.

In Example 8, the subject matter of Example 7 includes, wherein, toremove the ACKs from the transmission queue, the processing circuitryremoves ACKs from the multiple traffic flows in proportion to ACKscontributed by each of the multiple traffic flows.

In Example 9, the subject matter of Examples 1-8 includes, wherein theACKs are transmission control protocol (TCP) ACKs.

In Example 10, the subject matter of Example 9 includes, wherein theprocessing circuitry is to remove the ACKS from the transmission queueat a packet data convergence protocol (PDCP) layer of a cellularcommunications stack.

In Example 11, the subject matter of Example 10 includes, wherein aphysical layer of the network stack is a millimeter wave radio link.

In Example 12, the subject matter of Examples 9-11 includes, whereinmultiple TCP streams deposit ACKs into the transmission queue, themultiple TCP streams including a TCP slow start flow.

In Example 13, the subject matter of Example 12 includes, wherein ACKsfrom the TCP slow start flow are not removed from the transmissionqueue.

Example 14 is a method for uplink congestion mitigation, the methodcomprising: measuring a packet acknowledgement (ACK) rate in atransmission queue to detect when the ACK rate exceeds a threshold;removing, in response to the ACK rate exceeding the threshold, ACKs fromthe transmission queue in accordance with a time-based uplink reductionfunction; and transmitting ACKs that remain in the transmission queueafter ACKs are removed in accordance with the time-based uplinkreduction function.

In Example 15, the subject matter of Example 14 includes, wherein theACK rate is calculated by

${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$

where R is the ACK rate, T is a measurement window, d is a measurementinterval since a last sample, and r is an instant ACK rate.

In Example 16, the subject matter of Examples 14-15 includes, whereinthe time-based uplink reduction function removes a percentage of ACKs tomeet a percentage that corresponds to the threshold.

In Example 17, the subject matter of Example 16 includes, wherein thepercentage is expressed as a reduced ratio of ACKS to keep over ACKs toremove.

In Example 18, the subject matter of Example 17 includes, wherein ACKsare removed in accordance with the ratio.

In Example 19, the subject matter of Examples 16-18 includes, whereinthe time-based uplink reduction function includes a smoothing factor tosmooth ACK bursts over multiple uplink grant intervals.

In Example 20, the subject matter of Examples 14-19 includes, whereinmultiple traffic flows deposit ACKs into the transmission queue.

In Example 21, the subject matter of Example 20 includes, whereinremoving the ACKs from the transmission queue includes removing ACKsfrom the multiple traffic flows in proportion to ACKs contributed byeach of the multiple traffic flows.

In Example 22, the subject matter of Examples 14-21 includes, whereinthe ACKs are transmission control protocol (TCP) ACKs.

In Example 23, the subject matter of Example 22 includes, whereinremoving the ACKS from the transmission queue is performed at a packetdata convergence protocol (PDCP) layer of a cellular communicationsstack.

In Example 24, the subject matter of Example 23 includes, wherein aphysical layer of the network stack is a millimeter wave radio link.

In Example 25, the subject matter of Examples 22-24 includes, whereinmultiple TCP streams deposit ACKs into the transmission queue, themultiple TCP streams including a TCP slow start flow.

In Example 26, the subject matter of Example 25 includes, wherein ACKsfrom the TCP slow start flow are not removed from the transmissionqueue.

Example 27 is at least one machine readable medium includinginstructions that, when executed by processing circuitry, cause theprocessing circuitry to perform any method of Examples 14-26.

Example 28 is a system comprising means to perform any method ofExamples 14-26.

Example 29 is at least one machine readable medium includinginstructions for uplink congestion mitigation, the instructions, whenexecuted by processing circuitry, cause the processing circuitry toperform operations comprising: measuring a packet acknowledgement (ACK)rate in a transmission queue to detect when the ACK rate exceeds athreshold; removing, in response to the ACK rate exceeding thethreshold, ACKs from the transmission queue in accordance with atime-based uplink reduction function; and transmitting ACKs that remainin the transmission queue after ACKs are removed in accordance with thetime-based uplink reduction function.

In Example 30, the subject matter of Example 29 includes, wherein theACK rate is calculated by

${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$

where R is the ACK rate, T is a measurement window, d is a measurementinterval since a last sample, and r is an instant ACK rate.

In Example 31, the subject matter of Examples 29-30 includes, whereinthe time-based uplink reduction function removes a percentage of ACKs tomeet a percentage that corresponds to the threshold.

In Example 32, the subject matter of Example 31 includes, wherein thepercentage is expressed as a reduced ratio of ACKS to keep over ACKs toremove.

In Example 33, the subject matter of Example 32 includes, wherein ACKsare removed in accordance with the ratio.

In Example 34, the subject matter of Examples 31-33 includes, whereinthe time-based uplink reduction function includes a smoothing factor tosmooth ACK bursts over multiple uplink grant intervals.

In Example 35, the subject matter of Examples 29-34 includes, whereinmultiple traffic flows deposit ACKs into the transmission queue.

In Example 36, the subject matter of Example 35 includes, whereinremoving the ACKs from the transmission queue includes removing ACKsfrom the multiple traffic flows in proportion to ACKs contributed byeach of the multiple traffic flows.

In Example 37, the subject matter of Examples 29-36 includes, whereinthe ACKs are transmission control protocol (TCP) ACKs.

In Example 38, the subject matter of Example 37 includes, whereinremoving the ACKS from the transmission queue is performed at a packetdata convergence protocol (PDCP) layer of a cellular communicationsstack.

In Example 39, the subject matter of Example 38 includes, wherein aphysical layer of the network stack is a millimeter wave radio link.

In Example 40, the subject matter of Examples 37-39 includes, whereinmultiple TCP streams deposit ACKs into the transmission queue, themultiple TCP streams including a TCP slow start flow.

In Example 41, the subject matter of Example 40 includes, wherein ACKsfrom the TCP slow start flow are not removed from the transmissionqueue.

Example 42 is a system for uplink congestion mitigation, the systemcomprising: means for measuring a packet acknowledgement (ACK) rate in atransmission queue to detect when the ACK rate exceeds a threshold;means for removing, in response to the ACK rate exceeding the threshold,ACKs from the transmission queue in accordance with a time-based uplinkreduction function; and means for transmitting ACKs that remain in thetransmission queue after ACKs are removed in accordance with thetime-based uplink reduction function.

In Example 43, the subject matter of Example 42 includes, wherein theACK rate is calculated by

${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$

where R is the ACK rate, T is a measurement window, d is a measurementinterval since a last sample, and r is an instant ACK rate.

In Example 44, the subject matter of Examples 42-43 includes, whereinthe time-based uplink reduction function removes a percentage of ACKs tomeet a percentage that corresponds to the threshold.

In Example 45, the subject matter of Example 44 includes, wherein thepercentage is expressed as a reduced ratio of ACKS to keep over ACKs toremove.

In Example 46, the subject matter of Example 45 includes, wherein ACKsare removed in accordance with the ratio.

In Example 47, the subject matter of Examples 44-46 includes, whereinthe time-based uplink reduction function includes a smoothing factor tosmooth ACK bursts over multiple uplink grant intervals.

In Example 48, the subject matter of Examples 42-47 includes, whereinmultiple traffic flows deposit ACKs into the transmission queue.

In Example 49, the subject matter of Example 48 includes, wherein themeans for removing the ACKs from the transmission queue include meansfor removing ACKs from the multiple traffic flows in proportion to ACKscontributed by each of the multiple traffic flows.

In Example 50, the subject matter of Examples 42-49 includes, whereinthe ACKs are transmission control protocol (TCP) ACKs.

In Example 51, the subject matter of Example 50 includes, wherein themeans for removing the ACKS from the transmission queue is performed ata packet data convergence protocol (PDCP) layer of a cellularcommunications stack.

In Example 52, the subject matter of Example 51 includes, wherein aphysical layer of the network stack is a millimeter wave radio link.

In Example 53, the subject matter of Examples 50-52 includes, whereinmultiple TCP streams deposit ACKs into the transmission queue, themultiple TCP streams including a TCP slow start flow.

In Example 54, the subject matter of Example 53 includes, wherein ACKsfrom the TCP slow start flow are not removed from the transmissionqueue.

Example 55 is at least one machine-readable medium includinginstructions that, when executed by processing circuitry, cause theprocessing circuitry to perform operations to implement of any ofExamples 1-54.

Example 56 is an apparatus comprising means to implement of any ofExamples 1-54.

Example 57 is a system to implement of any of Examples 1-54.

Example 58 is a method to implement of any of Examples 1-54.

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration, specific embodiments that may bepracticed. These embodiments are also referred to herein as “examples.”Such examples may include elements in addition to those shown ordescribed. However, the present inventors also contemplate examples inwhich only those elements shown or described are provided. Moreover, thepresent inventors also contemplate examples using any combination orpermutation of those elements shown or described (or one or more aspectsthereof), either with respect to a particular example (or one or moreaspects thereof), or with respect to other examples (or one or moreaspects thereof) shown or described herein.

All publications, patents, and patent documents referred to in thisdocument are incorporated by reference herein in their entirety, asthough individually incorporated by reference. In the event ofinconsistent usages between this document and those documents soincorporated by reference, the usage in the incorporated reference(s)should be considered supplementary to that of this document, forirreconcilable inconsistencies, the usage in this document controls.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. In the appended claims, the terms “including” and“in which” are used as the plain-English equivalents of the respectiveterms “comprising” and “wherein.” Also, in the following claims, theterms “including” and “comprising” are open-ended, that is, a system,device, article, or process that includes elements in addition to thoselisted after such a term in a claim are still deemed to fall within thescope of that claim. Moreover, in the following claims, the terms“first,” “second,” and “third,” etc. are used merely as labels, and arenot intended to impose numerical requirements on their objects.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments may be used, such as by one of ordinary skill in the artupon reviewing the above description. The Abstract is to allow thereader to quickly ascertain the nature of the technical disclosure andis submitted with the understanding that it will not be used tointerpret or limit the scope or meaning of the claims. Also, in theabove Detailed Description, various features may be grouped together tostreamline the disclosure. This should not be interpreted as intendingthat an unclaimed disclosed feature is essential to any claim. Rather,inventive subject matter may lie in less than all features of aparticular disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment. The scope of the embodiments should bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

1. An apparatus for uplink congestion mitigation, the apparatus included in a communication device, the apparatus comprising: a buffer to store a transmission queue; and processing circuitry to: measure a packet acknowledgement (ACK) rate in a transmission queue to detect when the ACK rate exceeds a threshold; remove, in response to the ACK rate exceeding the threshold, ACKs from the transmission queue in accordance with a time-based uplink reduction function; and initiate transmission ACKs that remain in the transmission queue after ACKs are removed in accordance with the time-based uplink reduction function.
 2. The apparatus of claim 1, wherein the ACK rate is calculated by ${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$ where R is the ACK rate, T is a measurement window, d is a measurement interval since a last sample, and r is an instant ACK rate.
 3. The apparatus of claim 1, wherein the time-based uplink reduction function removes a percentage of ACKs to meet a percentage that corresponds to the threshold.
 4. The apparatus of claim 1, wherein multiple traffic flows deposit ACKs into the transmission queue.
 5. The apparatus of claim 4, wherein, to remove the ACKs from the transmission queue, the processing circuitry removes ACKs from the multiple traffic flows in proportion to ACKs contributed by each of the multiple traffic flows.
 6. The apparatus of claim 1, wherein the ACKs are transmission control protocol (TCP) ACKs.
 7. The apparatus of claim 6, wherein multiple TCP streams deposit ACKs into the transmission queue, the multiple TCP streams including a TCP slow start flow.
 8. The apparatus of claim 7, wherein ACKs from the TCP slow start flow are not removed from the transmission queue.
 9. A method for uplink congestion mitigation, the method comprising: measuring a packet acknowledgement (ACK) rate in a transmission queue to detect when the ACK rate exceeds a threshold; removing, in response to the ACK rate exceeding the threshold, ACKs from the transmission queue in accordance with a time-based uplink reduction function; and transmitting ACKs that remain in the transmission queue after ACKs are removed in accordance with the time-based uplink reduction function.
 10. The method of claim 9, wherein the ACK rate is calculated by ${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$ where R is the ACK rate, T is a measurement window, d is a measurement interval since a last sample, and r is an instant ACK rate.
 11. The method of claim 9, wherein the time-based uplink reduction function removes a percentage of ACKs to meet a percentage that corresponds to the threshold.
 12. The method of claim 9, wherein multiple traffic flows deposit ACKs into the transmission queue.
 13. The method of claim 12, wherein removing the ACKs from the transmission queue includes removing ACKs from the multiple traffic flows in proportion to ACKs contributed by each of the multiple traffic flows.
 14. The method of claim 9, wherein the ACKs are transmission control protocol (TCP) ACKs.
 15. The method of claim 14, wherein multiple TCP streams deposit ACKs into the transmission queue, the multiple TCP streams including a TCP slow start flow.
 16. The method of claim 15, wherein ACKs from the TCP slow start flow are not removed from the transmission queue.
 17. At least one non-transitory machine readable medium including instructions for uplink congestion mitigation, the instructions, when executed by processing circuitry, cause the processing circuitry to perform operations comprising: measuring a packet acknowledgement (ACK) rate in a transmission queue to detect when the ACK rate exceeds a threshold; removing, in response to the ACK rate exceeding the threshold, ACKs from the transmission queue in accordance with a time-based uplink reduction function; and transmitting ACKs that remain in the transmission queue after ACKs are removed in accordance with the time-based uplink reduction function.
 18. The at least one machine readable medium of claim 17, wherein the ACK rate is calculated by ${R = \frac{\left\{ {{R*{\max \left( {0,\left( {T - d} \right)} \right)}} + {{\min \left( {d,T} \right)}r}} \right\} T}{T}},$ where R is the ACK rate, T is a measurement window, d is a measurement interval since a last sample, and r is an instant ACK rate.
 19. The at least one machine readable medium of claim 17, wherein the time-based uplink reduction function removes a percentage of ACKs to meet a percentage that corresponds to the threshold.
 20. The at least one machine readable medium of claim 17, wherein multiple traffic flows deposit ACKs into the transmission queue.
 21. The at least one machine readable medium of claim 20, wherein removing the ACKs from the transmission queue includes removing ACKs from the multiple traffic flows in proportion to ACKs contributed by each of the multiple traffic flows.
 22. The at least one machine readable medium of claim 17, wherein the ACKs are transmission control protocol (TCP) ACKs.
 23. The at least one machine readable medium of claim 22, wherein multiple TCP streams deposit ACKs into the transmission queue, the multiple TCP streams including a TCP slow start flow.
 24. The at least one machine readable medium of claim 23, wherein ACKs from the TCP slow start flow are not removed from the transmission queue. 